Hybrid Forecasting Competition Data

Approved for Public Release; Distribution Unlimited. Public Release Case Number 20-2080

The Hybrid Forecasting Competition (HFC) was an Intelligence Advanced Research Projects Activity (IARPA) program to develop and test hybrid geopolitical forecasting systems. These systems integrated human and machine forecasting components to create maximally accurate, flexible, and scalable forecasting capabilities. Human-generated forecasts may be subject to cognitive biases and/or scalability limits. Machine-generated (i.e., statistical, computational) forecasting approaches may be more scalable and data-driven, but are often ill-suited to render forecasts for idiosyncratic or newly emerging geopolitical issues. Hybrid approaches hold promise for combining the strengths of these two approaches while mitigating their individual weaknesses. Performers developed systems that integrated human and machine forecasting contributions in novel ways. These systems competed in a multi-year competition to identify approaches that may enable the Intelligence Community (IC) to radically improve the accuracy and timeliness of geopolitical forecasts.
https://www.iarpa.gov/index.php/research-programs/hfc

Data are as follows:

Canonical_IFP_Bank_Dataverse.xlsx - spreadsheet that contains the domains, topics, and question templates delivered throughout the competitions.

HFC_Brier Score Calculations.docx - describes the Brier scoring method used throughout the competitions.

hfc-reports-codebooks.xlsx - descriptions of the files and metadata within the RCT folders.  


There are three sets of files corresponding to each Randomized Controlled Trial (RCT): 

RCTA - the first competition
Preseason - a short pre-test competition run prior to RCTB
RCTB - the second competition

These files are:

RCT_questions-answers.csv: A list of all questions (aka. IFP's), their answers, and associated metadata such as dates and descriptions.

RCT_daily-forecasts.csv: For each performer forecasting method, this report provides the last forecast for each scoring day (i.e., the last forecast before 2:01pm ET).

RCT_prediction-sets.csv: A list of all individual-level forecasts (aka. Prediction sets) created in the system.

=======
This publication is based upon work supported by the Intelligence Advanced Research Projects Activity (IARPA) [IARPA-BAA-16-02], via contract 2015-14120200002-002, and is subject to the Rights in Data-General Clause 52.227-14, Alt. IV (May 2014). Any views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA or the U.S. Government.  The U.S. Government is authorized to reproduce and distribute reprints for government purposes notwithstanding any copyright annotation therein.
©2020 The MITRE Corporation. All rights reserved.


